-
Notifications
You must be signed in to change notification settings - Fork 18.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend accurancy on multiple channels labels #759
Conversation
Ok it is a first tentative. I could collect some feedback |
@shelhamer Is this make sense? |
I still don't see the logic behind this. Could you pose an example ? On Saturday, July 26, 2014, bhack notifications@github.com wrote:
Sergio |
There are some cases where the source has N multi class label (on a single image). I can split the convolutional layer for feed N inner product layers (one per single label) and then I can use a N softmaxloss for every inner product. But I need to use the new slice layer on label blob to create N different top blob label. On test I want to calc global accuracy so I need to increment accuracy when N softmax layer predict exactly N label. So I need with a concat layer to aggregate back sliced labels and extend accuracy layer to support multiple labels aggregated in channels. |
@sguada @shelhamer Do you need some other info? |
@jeffdonahue if you are still active today can you take a look to this? I've some free time tomorrow and I want to fix it if it has any useful goal. |
Sorry, I'm not sure I understand how this is different from MultiLabelAccuracy by @sguada. Is this a generalization in that you're doing N 1-of-M predictions per datum, whereas MultiLabelAccuracy is for N binary predictions (1-of-2), and the current AccuracyLayer is for a single 1-of-M prediction? |
Yes exactly. I think this could be still "emulated" with @sguada Multi label "one hot" encoding using -1 0 1 to emulate every N softmax/softmax loss and then concat to feed Multilab accurancy. But i.e. we cannot use standard softmax/softmax loss to every single N and use standard accuracy. This try to enable this last behaviour. |
I guess there could be a case of use, but I think it is not very common. So As you noted one could either use several Softmax and Accuracy Layers, or Sergio 2014-07-30 1:02 GMT-07:00 bhack notifications@github.com:
|
@sguada An example that we already discussed in another thicket is partially in the schema at page 13 of this paper at the state of the art on SVHN dataset. |
I guess you can go ahead and try to implement such layer, but it seems a I don't see any problem using "one hot" emulation for that either. Sergio 2014-07-30 9:16 GMT-07:00 bhack notifications@github.com:
|
@sguada This PR tries to modify the actual accuracy layer it doesn't create a new layer. I'm waiting also for @shelhamer and @jeffdonahue feedbacks. If generally in this cases we want to handle this network design with "one hot" softmax emulation + label split + "innerproduct and label concat" to feed multilabel accurancy in #523 we can close this PR. |
I looked a little bit more in Multilabel accuracy in #523 and it doesn't work as a global accuracy on multitarget when we need to increment accuracy only when all target are predicted correctly and for one hot encoding use has still the problem of argmaxing probabilities. If we want to support a global accuracy from multiple loss we need to use something like this PR or extend the accuracy layer to support a vector of bottom blob of argmax=labels. |
I think having a global loss from multiple losses could be useful. So you Sergio 2014-08-05 14:04 GMT-07:00 bhack notifications@github.com:
|
Ok Sergio but I'm not talking about global loss from multiple loss. I'm talking of a global accuracy from multiple independent loss and labels. This PR extend Caffe accuracy layer try to cover this case. I'm using channels to concat labels. I could simplify this PR so that we could use in a standard way where argmax concat on channels == labels concat on channels. If we want to cover global accuracy with this configuration. |
I think the code you have right now, it is too specific. It assumes that all the set of labels have the same number labels. Therefore I don't see applicability to any other problem. |
I was not directly interested in modify the loss but i could try to work on this. What is your idea? If we have N single loss that try to optimize a branch of the DAG and contribute backward to a common parent (and this is already supported in Caffe) what kind o recombination of the single branches for a global loss are you thinking about? At informative level we need also a global accuracy so how we solve that? If as you tell is not generic to have argmax = labels how do you will handle the informative level? |
I see some general material here |
4278286
to
c01f07a
Compare
@shelhamer You have close the related ticket but what we want to do with this PR? |
Closing since the dev branch is deprecated. Please send PRs to master. |
With this I've tried to extend accuracy with multi channel label blob. Probably if it is fine test need to be extended.
Please take a look at #756